Truncated Approximate Dynamic Programming with Task-Dependent Terminal Value

We propose a new class of computationally fast algorithms to find close to optimal policy for Markov Decision Processes (MDP) with large finite horizon T. The main idea is that instead of planning until the time horizon T, we plan only up to a truncated horizon H ¡¡ T and use an estimate of the true optimal value function as the terminal value. Our approach of finding the terminal value function is to learn a mapping from an MDP to its value function by solving many similar MDPs during a training phase and fit a regression estimator. We analyze the method by providing an error propagation theorem that shows the effect of various sources of errors to the quality of the solution. We also empirically validate this approach in a realworld application of designing an energy management system for Hybrid Electric Vehicles with promising results. 2016 AAAI Conference on Artificial Intelligence This work may not be copied or reproduced in whole or in part for any commercial purpose. Permission to copy in whole or in part without payment of fee is granted for nonprofit educational and research purposes provided that all such whole or partial copies include the following: a notice that such copying is by permission of Mitsubishi Electric Research Laboratories, Inc.; an acknowledgment of the authors and individual contributions to the work; and all applicable portions of the copyright notice. Copying, reproduction, or republishing for any other purpose shall require a license with payment of fee to Mitsubishi Electric Research Laboratories, Inc. All rights reserved. Copyright c © Mitsubishi Electric Research Laboratories, Inc., 2016 201 Broadway, Cambridge, Massachusetts 02139 Truncated Approximate Dynamic Programming With Task-Dependent Terminal Value Amir-massoud Farahmand and Daniel N. Nikovski Mitsubishi Electric Research Laboratories Cambridge, MA, USA Yuji Igarashi and Hiroki Konaka Mitsubishi Electric Corporation Hyogo 661-8661, Japan